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1. INTRODUCTION 
— Monitoring biological characteristics 

Monitoring biological characterisitcs of the drivers such as electro-encephalo-gram (EEG), 
electro-oculo-graph (EOG), and electro-cardio-gram (ECG or EKG) that employ sensors to assess 
the tiredness of the person at the steering. The sleep of a person, to a large extent correlates with 
the functioning of the nervous system and provide useful inputs for identification of fatigue [1-3]. In this 
technique, the signals are captured through sensors fixed on drivers’ head, chest to check EEG, ECG and 
the like. The rate at which the heart beats during weariness exhibits distinct variations between diverse 
phases of attentiveness and exhaustion. As is evident for some of the research works that demonstrate the use 
of HRV that aids in measurement of weariness by emitting LF and HF signals between the range of 0.04 to 
0.15Hz and 0.14 to 0.25 Hz. Few research works show that drowsiness can also be measured using HRV 
which gives LF and HF signals, falls in range of 0.04-0.15Hz and 0.14-0.25Hz respectively [4]. 

Monitoring brain waves i.e., EEG has been a predominant instrument for assessing the tiredness 
of the driver. EEG waves have 4 frequency bands, delta band (0.5-4 Hz), which relates to sleep activity, 
the theta band (4-8 Hz), which corresponds to weariness, the alpha band (8-13 Hz), which indicate relaxation 
and creativity, and the beta band (13-25 Hz), which is indicative of alertness. An increase in the beta 
frequency band and decrease in the alpha frequency indicates drowsiness [5-11]. It has been said that the 
device NT-9200 for apprehending EEG signals in the varying degrees between the alert and the weary [12]. 
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The physiological indicators were instrumental in measuring the fatigue of the vehicle operator. However, the 
presence of the electrodes all throughout was irritating posing a setback to this approach. 

In yet another study, Xan Yu came up with an alternative solution to do away with the above said 
constraint wherein the steering of the vehicle is enclosed with a facilitating fabric to serve the purpose 
of electrode away from the driver. Further, the same has been replicated at the backrest of the driver’s seat. 
Though there was a favourable implications of the ECG signals so captured but what was a matter of concern 
was its effect on the seatback. The most noteworthy setbacks of this were enlisted to be the need for 
the driver to use both his hands on the steering for the capture of the waves, the failure of this method, lest 
the driver uses hand gloves and the effect of noise and its interference in the working of the system. 

— Tracking vehicle behaviour 

The behaviour of the vehicle can be used as a parameter for assessing the weariness of the driver, 
given the attributes of speed, curvature, experience on the wheels, and the state of mind of the driver. 
The different indicators of the vehicle performance such as the reliability of the standard deviation of steering 
angle and its velocity [13]. In yet another similar study, put forth the use of the rate of action of the steering 
such as the frequency and entropy, the prerequisites ranging from very small, to minor corrections, to very 
frequent oscillations and deviation from the normal to the discomfort of the driver and so on [14]. 

The deliverables of the safety system in question can be elevated at the assessment phase, working 
phase as well as the grass root level of the physical vehicle state. The entire onus is on tracking the weariness 
degrees [15]. As is discussed earlier, the speed at which the vehicle runs, the angle of the steering and the use 
of the brakes and accelerator are tracked by the vehicle data. Similarly the drivers performance is assessed by 
keeping track of the wheel movement, alignment to the bilateral lane, changing pattern of acceleration and 
use of brakes and so on. The use of sensors that are incorporated o the column of the steering wheels can be 
resorted to for obtaining the data for understanding the steering wheel movement (SWM). As has been 
reported, the micro-corrections of SWM were falling within the range of 0.50 and 50 [16-19]. 


2. SYSTEM BASED ON IMAGE PROCESSING 

It is an accepted fact that any system which tracks the physiological features along with the vehicle 
behaviour tends to be more dependable as the acquisition of the physical signals is relatively easier 
and reliable in terms of the results that it generates. However, what marks a drawback to otherwise very 
effective system is the fact that more are the number of signals, more is the difficulty in processing them thus 
making the system all the more complex and complicated. Another fact that cannot be undermined 
is the bulky size that defeats the importance of compactness for ease of installation in the vehicle. One more 
classification of the weariness of the vehicle operator detection system is that which is determined by 
the observation using image processing technique. The point to be reiterated here is that it does not make use 
of sensors to trace the driver signals rather tries to estimate the fatigue ness by capturing the data about 
the blinking of the eye. 

Research in this area has time and again affirmed that the fatigue of the driver starts with its effect 
on his eyes and mouth. Thus, the non-intrusive systems for fatigue detection takes into account the features 
of the face using computer vision approach. Facial Analysis technique is commonly used for many real-time 
applications such as for security checks in airports, in electronic gadgets for authentication, for legal issues 
and more so in surveillance systems [20]. It has been agreed upon that the most frequently used face 
recognition tools are eigen vectors, segmentation of skin, principal component analysis, template matching 
and artificial neural network. 


2.1. Face detection algorithms 

Face recognition is used in multidisciplinary fields like neural networks, pattern recognition 
psychology and image processing. The first break through was made in the 50s in the discipline 
of psychology. The findings drew attention towards face expression and emotional intelligence. 
But, the major research in this area was taken up in the 60s [21, 22]. Further, the researchers tried to extend it 
for human face recognition [23-26]. In extension of the same, they designed and implemented 
semi-automatic system. They came up with few issues that emerged up during their studies which acted as 
constraints for face recognition like brightness disparity, age factor, rotation and alignment of the head 
and face and expressions of the face, which surprisingly still exist despite of moving ahead 50 years from 
then [27, 28]. 

Research in 1970s led to the definition of geometric parameters and in depth study of pattern 
recognition in line of such parameters. To cite Kenade who attempted to design completely automated system 
for face recognition and found 45-75% accuracy in identification. Research in this domain continued in 
1980’s wherein the advocators propounded new approaches to enhance the existing systems like template 
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matching. The era of 90s mark the advent of systems relying on eigenfaces, PCA, ICA, LDA for 
face recognition [29]. 

Face detection as the name suggests involves acquisition of faces from the video stream and later 
is followed by extraction of regions, variations, alignment and spacing and then concludes with face 
recognition by involving comparison with image database [30]. In one of their studies, Yan, Kriegman a 
nd Ahuja attempted to classify face recognition on the basis of knowledge, feature, template and appearance. 
However, there is ever chance of these bases overlapping and combination of the same for developing an 
altogether new algorithm encompassing such categories. Enumerated below are the different categories: 

— Knowledge-based methods: recognizes face based on our knowledge of human faces. 

— System based on feature-invariant method: recognition of face in line with invariant method. 
— Systems based on template matching: comparison of input with stored attributes. 

— Systems based on appearance: identifies faces on the basis of training images. 

Knowledge based methods: these rely heavily on the shape and features of the face. Along with that 
they may also make use of symmetric eyes and the colour of eyes being darker than the skin beneath. 
Similarly, the distance between eyes and the intensity of colour in the upper and the lower segment of 
the eyes can be adhered to. Such models are economical in terms of computation cost but may have problems 
with rotation and alignment. Added to it there is risk of false negatives if the rules tend to be very general. 
These systems work one in simpler context and find it difficult to capture complex images. 

Feature-invariant methods: here, they resort to lower order attributes like corners, colour, shape and 
texture of the skin for detecting the face. The major drawback mentioned in the above model can be resolved 
here as it is rotation indifferent and scale independent. The capture time also is relatively less and hence 
the computation cost also comes down. However, the major setback with this is the choice of choosing the 
colour space, skin colour distribution model and the processing of the same [31-34]. 

Template matching methods: this is by far the most extensively used technique for detection as it 
tends to be unaffected by noise, is relatively faster and operationally more feasible that makes 
implementation all the more easy. This method falls back on the frontal face images that are predefined and 
stored in the database, with an algorithm being designed which correlates the input image with that of 
the stored one in the database and goes ahead with the detection. Despite of these merits, such algorithms 
face a limitation in terms of lighting and shapes of the face. A solution to this issue is to take an average from 
the face samples and store them and then to correlate the input image with the score and take it as the face 
position when it is the highest. This is otherwise known as the filter match method [35]. 

Appearance based methods: here, the onus is on statistical analysis and machine learning for 
detection of the required face attributes. The period from 1986 to 1990, saw the design of face recognition 
system by adhering to PCA whereby the face is synonymous with the coordinate system, the vectors being 
called eigenvectors. Further in 1997, new contributions were made in this direction based on neural network 
for face recognition comprising of two separate classes for face and non face attributes. However, the grave 
issue faced here was the training and representation of images that did not have faces. Similarly, support 
vector machines (SVMs) that are linear classifiers are resorted to as they maximise the margin between 
decision hyper plane and training set [36]. 


2.2. Eye detection algorithms 

A suitable and well defined and feasible technique for eye tracking was a major challenge for the 
researchers in the last decade. ET scans the manoeuvre of the eyes to predict the direction in which the driver 
is looking and the extent of time to which he looks. ET applications cover HCI, BCI, assistive technology, 
e-learning, psychology and like. Later it has been explained with the movement of eye during reading and 
narrated that the movements are not uniform and when using mirror they are not along the phrase. Much later 
to this an eye monitoring device was using contact lens that would gauge the pupil direction with a hole in 
the lens and the aluminium pointer attached to the lens helps in tracing direction. It has been observed using 
mirror that eye movements are not continuously along the phrase. Later an eye tracker device has been 
introduced using small contact lens with a hole for pupil gaze direction reading was done using an aluminium 
pointer connected to lens. 

It has been found with non-invasive eye tracking device, the first of its kind that relied on comeal 
reflection named photocronograph. Later its tried to trace horizontal movements using photographic plate and 
further it is designed with a photo device which recorded eye movements bilaterally [37]. The eye tracking 
systems encounter grave problems when the intensity of the pupil is weak and there is absence 
of brightness of the same. The tracing of the eye using IR lighting as the source calls for eyes that are open 
and unobstructed with glasses, hair or any other object, that are proximate to the camera with very less 
orientation under stable lighting and brightness. But such assumption are difficult to be met with as there 
would be constant changes in the brightness levels and blinking of eyes are but very natural and equally true 
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is the movement of head. Similarly, thick lenses of the spectacles distract the IR light thus making the pupil 
weak. The research spoke about the pupil detection method which ignores the background and sets threshold 
value to the minimum possible. In extension to it, Adhoc algorithm using threshold and morphological 
operations for elimination of glares on the glasses was resorted to with elimination of noise [38-40]. 

The study carried out was found that pupil tracking using eye appearance was effective provided 
there was no blockage to the eyes and the eyes were open. Further it has been observed that the real 
time subtraction using filter to do away with the illumination issues. Despite of this, the problem 
of obstruction of the eye still remains [41]. It has been found that the eye tracking method by amalgamating 
appearance-based methods and active IR illumination approach to have the synergy of both the approaches. 
However, obstruction, glasses and lighting conditions still remained the constraints. The study concluded that 
such systems can: (i) produce quality input images (ii) combine different complementary techniques, 
utilizing their strengths and overcoming limitations in order to detect pupil efficiently [42]. 

The traditional based methods in eye detection are further classified into three categories: 
- Template based methods 

- Appearance based methods 

= Feature based methods 

Template based methods: during ‘90s, researcher used template based methods for eye detection. In 
this method, a generic eye shape template is created initially. Template matching using correlation 
is performed on input image to find eyes. Here a method has been proposed for eye detection using accurate 
measurement spacing using hough transform (HT). This method is time-consuming, needs high-contrast eye 
images and it works only on frontal images. To improve the efficiency of template based methods, it has been 
proposed eye detection using deformable templates. In deformable template, first the eye model is allowed to 
translate, rotate and deform to fit the best representation of eye shape. This method can detect eyes accurately 
but they are computationally expensive and require images with good contrast [43]. 

Later the research says that the method based on template with optical flow with the success ranging 
to 88% on images and 73% on TV movies. However, the demerit of this system lies in the fact that it fails 
under face alignment issues and takes time to read each frame . It has been observed and propounded eye 
detection using deformable template by minimizing energy. The period of 2006 saw many studies that put 
forth eye detection using binary template grouping and SVM being tested on BioID database including 23 
images with a success of 96.8%, the problem of obstruction and closure of eye and the brightness still 
remaining the same. In response to it was the detections system using binary template matching with HT with 
an overwhelming success of 96.6% with the limitation of failure of capturing in case of using glasses [44-46]. 
Another work worth mentioning is the genetic algorithm (GA) with deformable templates that are extensively 
fall back upon by testing on the ORL database comprising of 400 images with the rate of success being 
87.2%. The major merit with this is its simplicity with relatively lesser mathematical calculations and works 
well provided the driver does not use glasses. 

Appearance based methods: as is likely from the name, such models capture eyes on the basis 
of photometric appearance. It calls for accumulation of huge database of training data showing diverse eye 
subjects, face alignment and lighting criteria. This is employed for training neural network or SVM. It has 
been found that the eigenvectors to categorise face, nose & eyes . The studies continued down the line in the 
1990s but mostly concentrated on frontal images [47]. The research shows attention towards eye detection 
with SVM classifier as the basis that was tested upon FERET databases with 450 images out of which 150 
images being frontal ones and the rest being equally divided into 33.750 rotated left & poses and had an 
overwhelming success of 96% without obstruction of eyes . 

The work says the use of eye detection based on SVM that was tested on local database with 97% 
success rate working under varying degrees of brightness but with a limitation of failure in capturing image 
in case of the driver using spectacles or face having different alignment. Moreover, the contributions 
propounded the utility of SVM & Zernike moments for detection of eyes and testing of the same on ORL 
database of 400 images with a success of 94.6% provided there is no obstruction of the face [48]. 
The contribution made an attempt with neural network with HT and tested on BioID with local database 
images of 304 having success rate of over 98% in contrast with the earlier methods with the condition of 
increase in the storage capacity requirement and elevation of the computation cost. That year saw the work 
has been identified in the same direction whereby it was tested on Olivetti Research Lab database with 75 
images therein having a rate of success of 88% despite of the varying degrees of light but is very complicated 
in terms of design and constraint of diverse background [49]. 

Feature based methods: as discussed above, this classification looks into the attributes like iris, 
corners, sclera in place of eyes. In this work, it has been suggested detection of the point as they found that 
such parameters are more static and hence easy to capture than eyes. Further the research work says that 
threw light on a model that enlisted six eye edges to be taken as points.Unfortunately, it did not succeed in 
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detecting closed eyes or those obstructed by hair or any other thing . Further it was propounded by a method 
wherein the eye model was supposed to be initialized in the first frame provided there was high contrast 
images to scan and trace eye edges [50, 51]. 

Similarly researchers propounded a system that relied on the vertical and horizontal projection for 
eye detection and concluded that the system fails to give the desired result if the eyebrows or eyelashes were 
not notably visible or were out of the image range and if the driver uses glasses. This takes approximately 0.6 
seconds for analysing the image. Similarly, the research says that the feature extraction using nonparametric 
discriminant analysis (NDA) & AdaBoost classifier that was tested on FERET database and came up with 
94.5% success rate [52, 53]. In has been identified that the technique whereby attributes like eyelid, look & 
face expression are captured and exhibited 96.4% success rate. Researchers, advocated a method wherein the 
driver is identified using iris texture and tested on iris database CASIA with the result being a rate of success 
close to 98%. 

As an evident from the works, though there was 88% success rate of the ORL database images 
technique method that made use of the eye detection with SVM, yet it failed to give necessary results in case 
of rotated images of the driver. Another contribution in this direction has been that the technique of eye 
detection that necessarily incorporates the template alignment merits with the features and exhibited 
favourable success rate of 95.2%. Yet the underlined drawback of this method was its inappropriateness if the 
driver used glasses [54]. As it is propounded by the analysis of the pixels of the image is costly in terms 
of time, effort and money when there is differences in the shape and colour of the face, they came up with 
a algorithm called Haar classifier to immediately pick upon objects based on Haar Feature rather than pixels. 

Researchers advocated the detection of the features of the face by resorting to Haar classifier, 
the same being tested on FERET database. The study concluded that the detection rate to be 95% while using 
1.2 GHz AMD processor. Few researchers used colour based segmentation to detect face and eyes. Similarly, 
the works propounded detection of the eye with the help of skin segmentation, which would otherwise be 
without relying on template matching and having detection rate of 98.4%. The algorithm has been proposed 
that uses template matching where a template is created for each image depending upon the data taken from 
the upper half of the face. Based on this, the part comprising the eyes is cropped and is taken as template for 
figuring out the eyes in the input image. It witnessed a success of 78% when tested on GTAV database [55]. 

Yet another study carried out the states that colour spaces can be used for face and eye detection. 
The said algorithm in question, works on the extraction of face by using the skin information. This algorithm 
tries to trace the skin region by changing RGB image to HSV which stands for Hue to give absolute colour, 
Saturation which talks about how white light interferes with the pure colour and Value that throws light on 
the intensity of the image. The algorithm demonstrated that the range of H lies between 0.01 and 0.1 for the 
skin. The algorithm witnessed a success of 95.2% when based on mathworks database images [56]. 

It has been identified that the utility of an algorithm that makes use of neural network perspective 
for eye detection by taking into consideration two phases namely the training for the same by making use 
of GTAV and VITS database images as well as the detection. It has been asserted that the working of this 
algorithm falls back upon the prerequisite of the training and the frequency of the images used for training. 
The more is the number the better is the outcome of the same. Two set of images are used for training the 
neural network are those comprising of object on one hand and the non-object images like the background, 
nose, eye brows etc. When tested on GTAV database, it exhibited a success of 98%. However, the inhibiting 
factor of this model has been the fact that it gives accurate results only on straight images and is unsuccessful 
in tracing both the eyes in case of 900 orientation of the face. However, when the alignment of the face is by 
300 to 450it captures one eye. Another major drawback of this algorithm is its inadequate ROI that results in 
the eyebrows being mistakenly identified to be eyes [57, 58]. 


3. ENCAPSULATION OF CONVENTIONAL METHODS 

Tracing of the eyes is done by resorting to noteworthy attributes like iris, colour of the pupil, shape 
of eye and its edges to mention a few so as to create a distinction between the eyes and other facial attributes. 
Unfortunately, to the dismay of the researchers, such methods don’t live up to the expectations due to 
the constraints posed by the illumination, face alignment and obstruction and winking and closing of eyes. 
Though to a large extent, the brightness problem is resolved using wavelet filtering but, still it is found to be 
useful on in case of slight variations in brightness and does not work efficiently if otherwise. 

A judicious blend of different colour conversion techniques can be employed for doing away with 
the illumination effects thereby leading to improved deliverables. Table 1 throws light on the diverse 
techniques for eye detection. The comparisons among the various eye detection techniques is done on 
the basis of fields, database type, frequency of images, technique involved, attainment statistics and 
enhancements and their constraints such as brightness, use of glasses, time consumption for capturing eyes, 
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face alignment and constraints posed by backgrounds in many of the cases. Although, there are notable work 
carried out in this field that demonstrates the elevated success rates, yet, the implementation of such system 
has its own fall-backs which have been mentioned earlier. Though the amalgamation of the two different 
approaches i.e., feature and appearance, would lead to the elevation of the suitability, feasibility and 
acceptability of the eye-detection in the actual parlance, but still it calls for rigorous efforts in this direction to 


do away with the drawbacks of the existing eye-detection algorithms. 


Table 1. Summary of eye detection techniques 





Total 





Image Success took seen 
Year Method Database No. of Rate (%) Improvement Limitations 
images 
Template based 8873 (TV Takes more time 
ie with optical flow ° : Movies) Beat Movement 2min/frame 
1998 SVM FERET 450 96 Rotated images Wearing Glasses 
2000 Contour i Stable against blinking, head Takes moretine 
approach translation, rotation 
2002 Edge Segment Local 120 90 Head Movement Wearing Glasses 
2004 SVM Local 7 97 Works for different Rotated images, 
Illumination wearing glasses 
2006 ae BioID 23 96.8 Rotated images Eyes Closed, Reflection 
Binary template 
aoe: “Matching & BiolD 23 95.6 Illumination BOttesimia bess 
Hough wearing glasses 
Transform 
2006 Haar Wavelets FRGC 1.0 - 94.5 Head movement Noise misleads eyes 
2008 ere ORL 400 94.6 Head movement Wearing glasses 
Deformable : ‘ : 
2009 femplate ORL 400 87.2 Less mathematical Wearing glasses, time 
fiatohing calculation consuming 
Fails if one or both eyes 
2009 ea e - - 90 Noise removal are closed. 
eo ES 15-20 sec on 2GHz 
2010 Line edge map Cal Tech 240 91.67 Illumination Wearing glasses 
Neunl Nework Yale ee 
2011 & Hough BiolD, 304 98.68 Search time is reduced i Be a 
tanehoen Local storage & high cost in 
computation 
a dak Illumination, uniform s 
Neural Network, Olivetti Stele Complex, change in 
any Wavelet research lab a Be ea background 
Doin .<, Rnowledse ewe 200 78 Ilumination Closed eyes 
based Template expression 
Skin MathWorks catia Works only on one 
20 Segmentation Video ms poe Boledaiaees database 
2012 Neural Network = GTAV 100 98 Tlumination Hose Toeson, 
background 
OOS eee 150 96 Rotation Ilumination 





4. CONCLUSION 


Based on the extensive review carried out on the already prevailing eye-detection techniques, 
it is evident that the parameters for an optimum technique can be enumerated as follows: (a) the setbacks 
of illumination effect can be vanquished by incorporating different colour conversion methods, 
(b) the synergy achieved by combination of diverse approaches would warrant a more reliable eye detection 
option, (c) algorithms on neural network analysis can come in handy for bringing down the search time, 
(d) algorithm should be capable of serving the purpose round the clock despite of the variations in 
the brightness, work for day/night time conditions under different brightness, clogged face and alignment, 
(e) there should be a provision for an alert system for cautioning the driver much in advance to avoid 
unwarranted accidents, (f) the complex design should be broken down into simpler ones, (g) The algorithm 
should be economical in terms of computation time. 

Weariness of the driver has been attributed to be the most frequent reason for catastrophic road 
accidents worldwide, which could otherwise be avoided provided necessary interventions are made on time. 
This holds good for the individual drivers as well as those in the logistics industry especially those who need 
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to drive for relatively very long monotonous distances without breaks. This is a socio-economic concern and 
can be ignored at the risk of the greater interest of the society at large. 

A closer look on the data of the national highway traffic safety administration (NHTSA) drives 
a point home that weariness of the driver is a potential contributor for road accidents and often elevates 
the risk of accidents by 5-6 times as compared to watchful drivers. Added to it, a majority of the road 
collisions take place at speeds above the safety limits. According to the reports of the world health 
organization (WHO), India ranks among the top in the world in terms of pathetic road conditions 
and the deaths caused thereby. The extensive research in this field throws light on the fact that the tiredness 
and weariness of the driver is the factor that has contributed to majority of road accidents. Fatigue, on one 
hand effects the alertness and the response intervention and at the same time increases the probability 
of being engulfed in the disaster on the other hand. Added to it, the driver with sleepiness fails to respond 
appropriately in case of a car crash, lest he could avoid the damage. 

Surprisingly, many a times, the driver may not be in a position to apprehend the degree to which he 
is tired and the point when he comes to saturation because of which he does not get a clue of the danger 
ahead. It is at this time, that the technology-enabled assistance would come handy to the driver so as to help 
him identify such symptoms. Such systems could be so designed to notify the driver using them in the event 
of loss of attention and warn them of the potential hazard awaiting them. 
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